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PREFACE 


Tliis  research  was  aucumplislied  under  project  7734,  Dcvelapmcpt  of  Methods  Tor 
Describing,  Eviiliiaiing,  and  Stnicturing  Air  Force  Occupations;  773401 16,  DevelDpnmnt 
unil  Nlodincation  uF  Coinprclicnsivc  OccupationiLl  Data  Analysis  ProgrLinis(CDl)AP). 

lliu  tiuihor,  an  ufUccrol'  the  Royul  Austfalian  At  Force,  is  curving  vvitli  tliu  United 
States  Air  Force  under  the  liKclinnge  Progrum*  1-LirUcular  appreciaUon  is  expressed  to  l^r, 
Raymond  E  Clirislui  for  his  guidance  and  advice. 

Tile  viesvs  expressed  in  this  report  are  not  necessarily  the  otTlcliil  views  of  Ihc 
United  States  Air  Force  or  the  Depnrtjnent  of  DercrLse. 
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COMPREHENSIVE  OCCUPATIONAL  DATA  ANALYSIS  PROGRAMS 
(CODAP):  USE  OF  REXALLTO  DENTIFY  DrVERGENT  RATERS 


L  INTRODUCTION 

REXALL  is  a  very  posverful  uiid  ncxlblc  prograni  within  the  Caniprchcnsive  Oecupiitioniil  Duta 
Analysis  Programs  (CODAP)  package,  being  designed  primarily  for  analysing  the  intcr-ralcr  iiyrecment 
among  judges'  task-fagtor  ratings.  One  use  of  REXALL  is  for  Identifying  divergent  raters.  At\cc  sonie 
general  eoinments  on  REXALL,  and  a  discussion  of  the  tenn  -^dlvorgcnt  rater,'*  this  rcp0rt  uses  data  from 
an  uctual  study  to  elaborate  on  ttiis  use  of  REXALL.  Tlie  decisions  made  on  the  basis  of  the  REXALL 
output  will  then  be  vorifled  by  reference  to  the  raw  data. 
' '      -    i/  '  •  ■  ■ 

IL  GENERAL  DISCUSSION  AND  BACKOROUND 

While  REXALL  was  designed  primarily  for  handling  task  faetor  ratings,  it  may  be  used  for  analysirig 
ancf  reporting  data  whenever  a  number  of  judges  rate  a  set  of  items  on  some  attribute.  For  example,  it  was 
applied  when  judges  rated  sets  of  officer  job  descriptions  on  various  atlributes  (CluistaU  1975),  and  when  a 
sample  of  officers  rated  a  set  of  education  profiles  on  educationial  suitability  for  service  in  a  particulnr 
utilization  field  (Watson  &  Goody,  1975).  It  may  be  applied  to  rankinp  as  well  as  to  ratings. 

The  items  rated  in  tlie  study  used  in  this  report  wore  the  SOS  tasks  in  the  task  inventory  for  the 
Medical  Service  Specialist.  A  total  of  93  first  line  supervlsots  rated  the  tasks  on  ^'Consequences  of 
Inadequate  Performance/-  The  instructions  page  froni  the  survey  booklet  is  presented  as  Appendix  A.  It 
contains  o  definition  of  the  task-factor  involved,  and  describes  the  scale  against  wliich  the  judges 

were  to  make  dieir  ratings.  The  9-point  scale  also  appears  as  Figure  L  Tliese  ratings  were  gathered  In  order 
to  obtain  a  measure,  the  mean  rating*  of  Consequences  of  Inadequate  Performance  for  each  task.  REXALL 
was  used  to  determine  whether  any  of  the  raters  should  be  deleted  from  the  study;  and  then  to  produce  a 
card  deck  containing  the  mean  task  ratings,  and  to  provide  a  meusure  of  the  intcr-rater  ap^cement  on  the 
ratinp. 


1 . 

Mlnlinal 

2. 

Slight 

3. 

Not  very  serious 

4. 

Fairly  serious 

5. 

Serloua 

6. 

Very  serious 

7. 

EKtremely  serious 

8. 

Almost  disastrous 

9, 

Disastrous 

Figtdre  L  9-point  rating  scde. 

••V  .  ■ 

A  general  description  of  REXALL  and  its  relation  to  other  CODAP  programs  has  been  documented 
by  Christal  and  Weissmuller  (1976),  Figure  2  is  a  schematic  presentation  of  the  inputs  to  and  outputs  from 
the  program.  The  outputs  describing  the  Items  (tasks)  that  are  rated  are  self^xplanatory  from  Figure  2, 
Both  outputs  describing  the  raters  .were  illustrated  by  Christal  and  Weissmuller  (1976).  The  Inter-rater 
Reliability  Table  includes  indexes  of  inter-rater  reliability  computed  by  the  Intraclass  correiation  formulas 
reported  by  Undquist  (1953).  The  Rater  Correlation  Table  (Example,  Table  1)  is  the  tool  for  detecting 
divergent  raters  and  will  be  treated  in  detail  later  in  this  report. 

Before  discussing  Table  I  in  detail,  the  term  "divergent  rater"  needs  clarification.  Tlife  is  siniply  a 
rater  whose  ratinp  are  substantially  different  from  diose  of  the  other  raters.  Tlie  most  common  of  these  are 
the  non-cooperative  raters  who  do  not  even  try  to  follow  die  instructions,  generating  instead  some  arbitrary 
pattern  of  responses.  Anodierlypc  of  divergent  rater  inverts  the  rating  scale-instead  of  rating  from  low  to 


TABLES  DESCRIBING  ITEMS  MTED 


TABLES  DESCRIBrNG  THE  RATERS 


Item  means  p  stand'= 
ard  deviations  and 
number  of  raters, 
(May  also  be  punch- 
ed as  card  decks.) 


Frequency  dlstrib^ 
utions  Eox  Item 
means  and  standard 
deviatiomi . 


INTER-MTER 
RELIABILITY  TABLE 


RATER  CORRELATION 
TABLE 


Figitm  2,  REXi'ML  inputs  and  outputs. 
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Table  h  Rater  CorrclQtion  Table  (In  Oripnal  Input  Order) 


Rater 

N 

SaniplQ 

ID 

Correlation 

Ratings 

Mean 

SD 

Mean 

T-Value 

1 

01 

.670 

505 

3.92 

1.00 

5,12 

20.22 

02 

.617 

446 

5.52 

1.93 

5.12 

16.53 

03 

.690 

504 

5.10 

1.13 

5.12 

21.35 

04 

.654 

484 

7.05 

1.19 

5.11 

18.98 

05 

.485 

503 

4.21 

.98 

5.12 

12.45 

06 

,589 

504 

7.54 

1,47 

5.12 

16.34 

07 

.667 

505 

5.62 

1.61 

5.12 

20.05 

08' 

,664 

305 

5.00 

l.U 

5.12 

19.91 

09 

.703 

505 

5.13 

1.62 

5.12 

22.18 

10 

.737 

505 

5. 10 

2.36 

5.12 

.24.47 

11 

,680 

502 

4.83 

1,33 

5,12 

20.74 

12 

.612 

504 

6.35 

1.35 

5,12 

17.35 

13 

.648 

505 

5. 18 

1.26 

5.12 

19.10' 

14 

.662 

496 

5.83 

3.12 

5.12 

19,64 

15 

,733 

503 

4.29 

1.58 

5.12 

24.11 

16 

.152 

505 

6,03 

1,85 

5.12 

3.45 

17 

,551 

504 

5.20 

2,17 

5,12 

15,19 

18 

.727 

504 

5. 18 

1.84 

5.12 

23.75 

19 

,544 

505 

.5,10 

1.68 

5.12 

14.53 

20 

,681 

504 

5.94 

1.56 

5.12 

20.86 

• 

■     %  ■ 

■  -  • 

f 

■  ■ 

■  :      "  .  . 

9 

9 

% 

• 

• 

• 

S 

9 

9 

• 

74 

.496 

503 

5.88 

1.34 

5.12 

12.80 

75 

.547 

505 

6.52 

1.60 

5.12 

14.66 

76 

.511 

480 

4.64 

1.25 

5.15 

12.99 

77 

,474 

:  503 

5.41 

1,01 

5,12 

12.05 

78 

.704 

505 

4.92 

1.00 

5.12 

22.24- 

79 

.553 

500 

6.29 

2.06 

5.13 

14.81 

80 

.707 

501 

6. 14 

1.11 

5.11 

22.32 

81 

,480 

502 

8.21 

1.74 

5.12 

12,23 

82 

,727 

504 

5.92 

1.70 

5.12 

23.75 

83 

,523 

150 

1.99 

2.00 

4.66 

7.47 

84 

,474 

95 

4.71 

.71 

'  4  .87 

5.19 

-,115 

505 

1.02 

.15 

5.12 

-  2.61 

■86 

.567 

294 

2.61 

1.66 

5.19 

11.77 

87 

.000 

505 

9.00 

,00 

5.12 

.00 

88 

.083 

96 

6.77 

1.39 

4.56 

.81 

89 

.70S 

505  . 

6.72 

2.06 

5.12 

22.50 

90 

.705 

503 

4.73 

.89 

5.12 

22.25 

91 

.518 

503 

4.50 

■  .64 

5,12 

13.55 

92 

.562 

504 

3.87 

.69 

5,12 

15.24 

93 

.554 

496 

3.10 

1.08 

5.12 

14.80  ' 
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liigli  tlic  ratur  rntcs  fium  high  to  bsv.  As  will  bu  dunionstrulotl  sliortly,  both  iy\m  arc  CM^IIy  td^ntiricrf  Irom 
REXALL's  Rater  Ci)^;rulatiun  Table.  Tliu  nuincoDix^railve  mlcr  Is  iisunlly  droppecl  fmm  1N  study,  luu! 
inverted  ratings  can  be  arithmeticaliy  reversad  ar  drnp|)ad  from  Ihc  study  ut  tliy  optlorii  «f  ilH*  rcsuurchcr. 
Yet  niinther  lype  of  dlvergciU  rater  is  iiol  ho  cluar  eut,  Tills  is  the  rater  wliosc  |)crceptton  uif  Uie  ituniS  being 
rated  different  from  iliat  of  ilie  other  niters,  TliL^  difference  In  pcrcoptlon  may  be  \^^ny  timl  or  may  In 
affect  be  a  lack  of  discriniinatiun  posyeror  of  knowledge  on  tlie  part  of  Uic  rater.  In  imy  m^s  Uic  decision 
on  whether  or  not  to  retain  such  a  rater  in  the  study  is  rutlicr  sulijcatlvc  and  each  (:i*sc  ma^^l^g j\idpd  on  Us 
merits. 


UU  USINGT1II2  RATUR  CORRHLATtONTABLE 
TO  IDEmriFY  DIVURGENT  RATl£liS 

Tlio  Rater  Correlation  Tabic  produced  by  REXALL  contiuns  a  row  of  data  for  ea^h  mllor,  prlnlad  In 
lliti  same  order  in  svhidi  thy  cmc%  sverc  input  to  tlie  program.  Taljto  1  is  an  extract  (mm  tK^  tabic  for  the 
Illustrative  study  being  used  in' tHis4cpc\f1,.raters  2 1  through  73  being  omitted  because  «f$'paQ^  limitaUons, 
Tlie  correUition  eolunin  contains  tho  correlations  between  coch  ruler's  ratings  and  tlia  rriMiainii  for  aJI  raters 
on  the  same  tasks,  the  extreme  right-liand  column  being  a  Value''  for  delermining  ilie  lavel  of 
significance  of  deviation  of  this  correlation  coefncient  from  ^ro,  Tlie  other  four  coluiru^i  m  the  number 
of  Itttnis  (tasks)  rated  by  each  rater,  the  rater's  mean  and  standard  deviation  and  the  "^ampl^  moan.'' The 
sample  ineaii  is  tlie  mean  of  all  raters'  ratings  on  the  tasks  rated  by  thnt  rater. 

It  is  this  table  that  is  used  to  decide  wliich  raters  arc  divergent  and  whether  or  not  tNy  should  be 
deleted  I  rom  the  study.  Tlie  following  paragraplis  describe  how  diesc  decisions  were  ^t,iWtsWfcn  tho  study 
being  used  to  illustrate  this  report,  Foroase  of  CHplanatlan,  die  table  entries  containiog  th^  JO  higlieit  und 
the  20  lowest  correlations  have  been  extracted  ftom  the  oripnal  table  and  are  presonladi  w  Table  2.  Wlien 
the  program  is  next  revised,  it  Is  planned  to  pro^dc  an  option  to  repeat  fablo  1 ,  rearr9;rt5qd  descending 
ordcrof  correlation  coefncient,  and  also  on  other  variables 

From  Table  2  It  is  evident  that  the  majority  of  tho  gorrelatlon  coefndents  exceed  tmU  77% 

cxgeed  .50).  Even  the  '*good"  raters  vary  in  number  of  tasks  rated  and  in  mean  and  st^nd^jfd  deviation  of 
ratinp.  However,  their  high  correlations  wilii  the  group  indicate  that  Uiey  share  similar  jmc^t^ilofi^of  the 
relative  consequences  of  inadequate  performance  of  the  tasks  In  the  inventor'. 

Tlie  prime  indicators  of  divergent  raters  are  the  correlation  coefricicnts.  A  lii^i  neptiv^  eoefricicnt 
indicates  inverted  ratings,  while  an  insignificant  coefncient  is  the  result  of  a  non<!OOp<^'mt:lvp  rater.  Tlie 
raters  who  have  made  a  genuine  attempt  to  obey  the  instructions^  but  whose  perceptions,  oflh'P  talks  differ 
from  those  of  the  majority  of  the  raters,  should  have  a  rclulively  low,  but  signiflqiuvt,  correlation 
coefficient.  Unfortunately,  this  condition  may  also  typify  the  rater  who  cooperatos,  tm  part  of  the 
inventory  and  then  becomes  non-cooperative. 

Table  2  will  now  be  used  to  identify  ilie  divergent  raters  In  this  study,  Tliere  does  fioi  appear  to  be 
any  cases  of  inverted  ratings.  Hie  last  six  raters  (In  Table  2)  are  apparently  non-cooporgtiv^  ir^tors,  and  at 
least  the  next  two  require  social  eonsideratlon  as  their  perceptions  of  tlie  tasks  seem  to  be  Jiifrerent  from 
those  of  the  majority  of  the  raters.  Each  of  the  last  eight  raters  in  Table  2  will  be  discussed 
individually. 

Rater  85,  The  correlation  coeffiglent  (-*11S)  could,  in  isolation,  suggest  Inver^c^d  tatSfigs  but  the 
other  data  available  suggests  otlierwise.  All  tasks  were  rated.  Tlie  rater's  mean  of  (,03  ftnd  standard 
deviation  of .  1 5  suggest  that  almost  every  task  has  been  rated  1 ,  witli  only  a  few  higlier  ratlMp,  Probably  by 
chance,  these  higlier  ratings  were  allocated  to  lower  Uian  average  tasks,  giving  rim  to  thq  rmgativo 
correlation  coefncient.  Rater  85  appears  to  have  been  non-cooperative  and  was  deleted  fifpfn  th^  study* 

Rater  4],  A  mean  of  1 .33  (standard  deviation  3\)  indicates  most  tasks  were  rated  \  wiilii^i sprinkling 
of  higher  ratings,  pediaps  just  to  break  up  the  pattern,  Rater41  seems  to  be  another  noivcoopmtive  rater 
aiid  was  also  dropped  from  the  study, 

.    Rater  St  Tlic  mean  of  9.00  with  a  zuro  standard  deviation  indicates  this  rater  rated  «V^ry  tosk  9,  Thh 
is  perhaps  the  most  obvious  type  of  lack  of  cooperation.  Rater  87  was  deleted. 
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Table  2,  mm  CwrolatSon  Tabic  (Ilcofdercd  from  lli^i  la  Low  Correliitloii) 

— .    ^—  .j.:-^-7;:^_:_-_jjLLijfy_UMi!<-j!MTiW^  .j       —  -  -  i 


Rator 
ID 

Corroliitiwtt 

N 

Rauinga 

Moan 

SD 

Sflinple 
Maan 

T-Vq1ug 

67 

.764 

505 

3.07 

1.43 

5.12 

26.54 

10 

.737 

505 

5,10 

.  2,36 

5,12 

24.47 

15 

.733 

503 

4.29 

1.58 

5.12 

24 . 1 1 

82 

.727 

504 

5.92 

1.70 

5.12 

23,75 

18 

.727 

504 

5.18 

1.84 

5.12 

23.75 

35 

.726 

503 

4,49 

1.44 

5.12 

23.66 

44 

.720 

503 

4.81 

1.52 

5.12 

23. 23 

65 

.720 

494 

4.19 

1.07 

5.12 

23.00 

89 

.  708 

505 

6.72 

2.06 

5.12 

22.50 

80 

.707 

501 

6.14 

1.11 

5.11 

22.32 

90 

.705 

503 

4.73 

.89 

5.12 

22.25 

78 

.  704 

505 

4,92 

1.00 

5.12 

■  22.24 

09 

.703 

505 

5,13 

1,62 

5.12 

22.18 

21 

.703 

502 

4.86 

1.29 

5.12 

22.10 

60 

.699 

479 

4.41 

1.40 

5.14 

21.35 

72 

.695 

.  505 

4.04 

.84  , 

5.12 

21.69 

34 

.694 

503 

4. 73 

1.13 

5.11 

21.60 

03 

.690 

504 

5.10 

1.13 

5.12 

21.35 

64 

.  689 

375 

4,72 

2,31 

5.23 

18.34 

53 

.687 

505 

6.13 

1.70 

5.12 

21.18 

t 

• 

■ 

• 

1 

• 

• 

• 

• 

« 

• 

• 

0 

• 

• 

• 

• 

• 

• 

» 

• 

74 

.496 

503 

5,88 

1,34 

5.12 

12.80 

05 

.486 

503 

4,21 

.98 

5.12 

12.45 

81 

.480 

502 

8.21 

1.74 

5.12 

12.23 

77 

.474 

503 

5.41 

l.Ol 

5.12 

12.05 

84 

.474 

95 

4.71 

.71 

4.87 

5.19 

31 

.454 

479 

8.44 

1.14 

5,14 

11.12 

27 

.439 

504 

5.50 

2.06 

5,12 

10.96 

24 

.396 

502 

3.62 

1.25 

5,12 

9.64 

52 

.377 

505 

2.41 

2.48 

5.12 

9.12 

48 

■  .358 

505 

4.98 

.94 

5.12 

8.60 

68 

.327 

505 

3.93 

.51 

5.12 

7.76 

63 

.245 

384 

3.02 

2.64 

5.09 

4.94 

7  70 

J  9  Id. 

16 

.152 

505 

6.03 

1.85 

5.12 

3.45 

61 

.106 

505 

5.14 

.96 

5.12 

2.38 

88 

.083 

96 

6.77 

1.39 

4.56 

.81 

22 

.047 

505 

2.69 

2,09 

5.12 

1.07 

87 

.000 

505 

9.00 

.00 

5.12 

.00 

41 

-.012 

505 

1.33 

.81 

5.12 

-  .26 

85 

-.115" 

505 

1.02 

as 

5.12 

-2.61, 

■  ■■       ■     .  9 

ER?C 


Ranr  21  'tills  cuniblnutiLm  of  very  ly\v  corfolntloii  C047)  Jew  nicmi  (2,59)  and  lueh  stmdiifd 
duvialion  {2,09),  when  the  ralcr  1ms  responded  to  uvcry  tiiBi,  su^gcstiaprDiieimity^ofl  ora  riitings,wltlio 
numbar  oF  higher  ratings  (soim  up  near  thu  top  the  st^alc)  uiloealed  intllscriniiiiatoly  iimoiig  the  olhor 
tusks*  Rater  22  was  troatcd  as  a  noivcoupcrativc  ratar  at|d  delotcd* 

Ram  88.  mis  rater  responded  to  only  19^)  wf  the  tpjs.  MtJitHigli  thi^  19^8  af  the  taski  imimi  to  be 
less  dcmiifidirig  Ihan  average  (Humplc  mean  -  4.S6;  ovornll  mm  -  5  ,12),  (his  mtcravcrnged  6.77  Dntliem- 
Tlio  ml\i\^  thnt  svcre  pravidcd  tiru  therelbra  vtfiy  hifeli,  iind  ii  ijorrdatbil  of  .083HUgycMs  they  arctiot  vcr>/ 
roaliHtie.  The  msigificant  carrulation  and  low  nuinberof  tusks faM  Justiiy  delciirig  tills  niter  on  tlic  grpunds 
urlack  of  cooptfrptlDn,  - 

katerei  l\m  case  npproachcs  the  dgubtfiil  zoflc.  'Ilic  darrdallcu  Is  loo  low  to  beliyvo  the  rater  has 
been  entirely  uaoporativc;  tmd  it  Is  too  low  to  n^umy  asiinplcdlFfoicricelnporc^'n^  tajks.  Rntcr 

61  wus  dropped  from  the  study  as  being  non-^cooporativc. 

Rater  16,  Apart  from  the  relatively  low  correbliori  coefncicntj  this  rateri  stnlistii^  seem  falrlj^ 
nonnal,  wm  that  the  mean  is  perhaps  a  little  higlu  Tlieresecin  to  be  three  possibUUies:  there  iseitJmrti 
geniiino  diiTorence  In  ijorccption,  or  a  lack  of  discriiniridtiori  po^er,  era  lack  of  □omplctccoo!)aratlQiu  A. 
clear  decision  cajinot  be  made  on  the  data  aVaibble,  Aj  tJiere  m  plenty  oF  raters,  and  this  one  repTeseati  an 
isolated  opiniori  if  the  ratings  are  genuine,  the  analyst  chos&  to  drop  lilin  from  the  study  even  thou^i 
reten  lion  woidd  mi  liave  seriously  affected  the  talk  mom  ard  the  inter-rater  reliability  statistics, 

Raicr  S4  Tlic  analysis  for  this  rater  is  very  parallDl  to  that  for  Rater  16,  Ibwever,  the  very  hi^  meaa 
(7.79)  and  standard  deviation  (2.03)  Indicates  a  veiy  lajgeriurTiberof  ?  ratmgs,  perhaps  genuinely  bolioved 
to  bo  jusiined,  Tliis  rating  pattern  has  depleted  the  mtcr-s  dUcriiniiiatjon  power  at  the  lii|i  end  of  the  icalo 
and  thus  cdysod  the  relatively  low  corrclatioti  coefndeiit  Tito  declsbn  on  Mietlier  or  not  to  delete  this 
rater  should  have  Utile  effect  on  the  objectives  of  tli&  study.  Wliile  anequally  pod  case  could  be  inud€R)r 
deletion,  the  analyst  chose  to  retain  tills  rater  in 

^  Tlie  next  six  to  ten  niters  above  Rater  54  In  Table  2^^lso  have  npp  rcdably  low^cr  cDrrelationg  than  the 
majority  of  raters  mid  could  be  classed  as  ajvergcnt  raters.  However,  their  ratiags  are  considered  geiiiilrQ 
■  and  the  lowqr  corrdutions  can  usually  be  explained.  For  eWiplo,  the  analysis  for  Rater  SI  ^ould  piiraller 
that  for  Rater  54;  Raters  53  and  52  are  the  same  except  that  by  ravorang  tlio  low  cad  of  the  scale  tliey  liave 
deploled  dicir  djsqTiminntion  at  tliat  end  of  the  scale,  Riiteri  58,48,  and  84  deinomstrate  their  low 
discrimlnution  power  by  tlieir  lack  of  variation  in  ratJap.  As  linplicd  hy  the  aaalysis  For  Rater  5^,  more 
than  seven  raters  could  have  been  deleted  witl^out  sigmfitaatly  afrcctlrig  the  objectives  of  the  stady  either 
way,  Wliero  to  '"draw  Uia  line"  must  remain,  for  the  tliHe  being,  a  siibjectivu  dedibn  m  the  part  of  the^ 
analyst  One  niar0  objective  approach  being  eKanilnDd  is  to  have  tlie  program  progressively  clinunate  the 
most  divergent  rutors,  one  at  a  time,  until  the  Inter^Tater  reliability  statistic  for  the  stability  of  theltetu 
(task)  meani  (Rj^j^)  ceases  to  Increase. 

At  tilts  point,  one  further  comment  ori  divergent  raters  should  be  iiiEde.^s  is  seen  from  Tables  I  and 
2,  there  are  conildprable  differences  among  the  raters*  rnean^  and  among  their  standard  deviatioaj,  caused 
by  dirrereiU  perceptions  of  the  words  used  to  describe  the  levels  on  the?.point  rating  scale,  Hovv^cver,the 
magnitudo  of  the  correlations  for  the  cases  retained  it^  tjifc  study  are  satisfactory  cadence  tlial  these  rate« 
shared  sufflciemly  fjrnilar  perceptions  of  the  relative  consequences  oririadcquate  perforniance  of  the  tasks 
In  the  inventory.- As  it  is  a  measure  of  the  relative  consecjuorcei  tliat  is  required, no  xater  was  declared 
divergent  on  the  grounds  of  a  high  or  low  mean.  In  ftct,  tlicie  difftroiices i n  iTiea!is  mid  standard  deviations 
only  add  withlit  talk  variance  tliat  is  not  justified  when  relative  measures  are  being s ougiit  ^Accordingly  the 
standardization  option  discussed  by  Christd  and  WelssmuUler  (l91&)  ms  used  in  this  study  to  remove  tlie 
between  rater  variance. 

Tlie  effects  of  delctirig  the  seven  diveiieiit  raters  is  presented  In  Table  3,  Deletiiig  the  divergent  raters, 
the  ones  who  did  not  cooperate  and  those  who  perceived  the  relative  valiics  of  tlic  tasladifferciitly^does 
approdably  increaso  the  inter^rater  reliability  and  therefore  increasei  the  stability  of  tlie  task:  means!  Tliis 
applies  whether  or  not  the  data  are  standardized. 
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Tabk  3.  Efficts  of  Deleting  Sev«n  Dlverscnt  Rafc^rs, 


Raw^  ratipp,  Odelofioiis  93  MSl  J41 

Baw^  ratirgi,  IdeJetioni  S6  .195  JS2- 

Sjaiidardized  ratiris,  O  detetioiis  .  93  ,313  J7S 

'Staiidardlzed  Atbip/ 1 B6  ,353 


IV.  YERIFIOTICN  OF  DECI^IOHS  BTf  RBrE»EN^^ 
TC  IUVW  JDhTA 

Havinimade  tlia  above  decldciii  basad  ori  tha  REXAlL  raw  data  f^reach  rsier 

was  obtained  to ixaniinMhdr  validity,  hmamof  the  ta^  ratirgi  are  preie:^ted  In  ^ppandbKOi  B,C^andD 
as  blocks  of  rav/ railnigii  ia<h  blocK  contatolng  all  the  ratiapmsde  by  cnt  rater.  Each  blb^  QQ.rilains eight 
rows  of  di^ts,  69  digits  in  each  of  the  flrit  sev^n  rows  and  22  In  tiie  lait 

one  for  eadi  ink  m  ihQ  inymtory.  Each  ddgit  is  a  laslcrating  by  the  r^ler  lo  \^Ndi  ih§  W-cqIc  mfeHj^aO 
indicating  the  rater  did  not  rate  the  %f  on  th&  first  row  Is  ttio  rating  for  T^ft  1^  the  last  on 

that  row  is  Task  69,  and  so  en*  tlie  l^st  digit  on  tlie  last  *ow  lelngth'^ratijig  cnTosIc  505, 

^  /  Appendix  B  isthe  ratangs  for  lhre€''gcod*' ratori,_  provided  foJcofiip 
that  the  raters  have  consldafed  and  rated  each  4a^  Individ nally,  T^ili  does  aiot  mesnj  ofco^tig  ,that  tli^lr 
ratings  are  Identical,  Bccaxise  of  differeiicai  In  OKperienceV  iv©^^^  ate.,  there  wil3  be  lonie 

differenceUn  perceptiofi;  arid  of  courie  %^  lonriMrJOr^arlafise. 

;  In  contrast  with  Appondijc  B,  Appesidlx  C  li  tW  raw^^ratinigi  fcr  ttie  six  rglers  ass*cise^l  as  belJig 
non-cooperativa/Each  vfQl  now  be  dlsciiss^d  iii  turn,  rafleciini  o*  thte  di^gnases  prevlQusly  niad^  fr^in 
Table2(thaREXALL©utpiit).  v 

JRater  85^  As  predictgdi  RateriS  late^l nearly' ^veiy  tss  l,rao  seven  2ratlngsaadori^3  rotiniware 
all  allocated  to  tasks  %^1i1q1x  most  raters  oonMered  less  oorise^iJDntSal  tJan  average.  For  exaniple,  the  oiie 
task  that  was  rated  3  was  ^^scheduleleavgi  or  passes'*  This  ishardly^as  consequemialai*  for  iCHaniplaitasta 
involving  rendering  emerpncy  tmtrn^  BS, 

Rater  4L-  The  prediotion  was  that  m^it  &f  this  rater*s  Tatliig&  wotild  bo  1  wltlia  si)riiiI<iirigof  lilgttof 
-  ratings,  Notice  the  tliree,9  ratings.  Tliescthree  t^skswero:  (g)  plan  rCgordi  imaintouaaiaes  (b)  direct  physical 
exercise  or  conditlonini  programs^  and  (c J  direct  preparatiori  mi  maintenance  of  records  or  reports. 
Considering  the  liuniber  of  lasksin  the  inventoo'that  cculdresiilt  itith^lmi^^  ap^tient.it  li 

evident4his  rater  Was  nonwoperativo.  Pcha  eHerelie  Jilis^nse  oftiumcr, 

Rater  2^.  A  propensity  of  1  and  2  ratings  vwai  ^jredlctedj^lth  ionne  hlghL  ratflngs  allecat^d 
indiscriminately,  This  diagnosis  is  coiiflrnied/ Tlie  indlicrltiinate  nature  of  tlie  Wgh  rospQuses  ean  be 
illustrated  by  coriiidcringtw^o tasks  fro  acrcmodieadeyaDuatlQrvduio':(a)432/niake 
;  434.  operate  inflight  emerge  Jicy  oxyien  systems^  Th^  mean  overall  ratingson  these  two  tasks  WQffi3.56  and 
_6-30  jespectiyely.  but  this  r^ter  rated  the  nr^i  task  ''alitmoit  JisastroiJf  pn  SD^an^  tMiecondl  ds  ^'minlinil" 

'];(a"ijrrtjS'RifoK^ 

did  notcomply  with  the  Instructions. 

^gftfr  The  raw  ra  coiinmt  tho  analy'sis  from  the  daia  in  Table  2.  A.  faLnl-h^arted  atiDnift 
seems  to  have  been  made  at  the  early  tasks  and  the  rater  has  left  the  rest  oftlie  b  ooklet  bljinfe.TTnii  pnltefn 
is  fairly  common  atnong  noH'Cooperative  racors  who  seam  t  o  believe  no  on@  wLlJ  ever  dcigcHiielr  la(ik  of 
cooperation  XtJMswr^^y  bocklet  appoa  ' 


;  Rarh  61,  The  pattern  of  this  rater's  raw  responses  explains  his  low  cotielation.  Hie  ratings  for  the 
first  71  tasks  aie  my  consistent  with  those  of  the  group.  Howavefs  at  that  point  the  rater  has  became 
nori-cooperatwe  and  rated  aU  but  one  of  the  remaiiiing  tadcs  5.  To  examine  how  good  this  imter's  first  71 
responses  were,  the  labia  entries  that  woxildliave  resulted  had  tasks  72  through  505  not  been  rated  at  all, 
ware  QOniputed^  They  appear  as  Table  4.  These  statistics  are  quite  acceptable*  It  would  have  bean  far  better 
if  Rater  61  hod  left  these  remaiiiing  tasks  blank. 


TabM:  Rater  CorrelationTaWe  En 
fox  Rater  Number  61  with 
Tasks  TTiated  as  Blank 


Correlalion  • 

=  .637 

N  Ratin|s  = 

"  71 

Mean     •  « 

6.03 

Standaid  Deviatloii  ' 

*  2.34 

SamplftMean  = 

4.S1 

T-Value 

-  6.87 

.  The  raw  data  for  the  tv/o  bordeiline  cases  ^tei  16  and  54)  are  preieiited  in  Appendix  D,  As 
.  sugpsted  In  the  earlier  analysis^  their  ratirigs  diould  appear  fairly  nomal  unless  they  were  partially 
cooperative  rateTs  (like  Rater  61).  Except  for  the  atribfriially  large  number  of  high  ratings,  part^ 
Rater  54,  both  i^tliig  patteriis  seern  reasonable,  As  already  mentioried,  the  decisionon  whadier  to  accept  or 
rejeet  those  two  raters  is  rather  subjective,  eitWr  course  having  little,  effeet  on  the  mean  ratings  of  the  talks, 
^ater  54  has  giv^en  more  vdid  ratinp  than  Rater  16^  although  they  are  far  from  perfect,  There  seems  to  be 
iiD  riaion  to'cliaJngethe  previous  decision  which  was  to  accept  oneand  reject  the  other,  V  = 

I  V,  CONCLUSION 

All  tlie  decisions  regarding  divergent  raters  in  tMs  studyj  rrade  on  the  basis  of  REXALL's  Rater 
Correlation  Table,  have  been  verlfled  by  rafereii^^to  the  raw  data.  TTiere  were  no  cases  of  inverted  ratings 
in  this  study,  altliov^  arithmetical  inversion  of  audi  ratinpin  another  itudy  conducted  by  the  author  did 
testify  to  the  validity  of  treating  jughneptlve  correlatiena  as  indicators  of  scale  re^enal.The  interpretation 
of  relatively  low,  but  significantj  correlatloa  coafficients  must  remain  some\v»hat  slibjecti^^  for  the  time 
being.  More  objective  approaches  to  the  liandlijig  of  such  cases  are  currently  under  coiisl  da  ration  by  the  Air 
Force  llurnqn  Resources  Laboratory.  ., 


Christal,  B^E.'  SysMtnatic  method  for  estabiishin^  cffwm  grade  rcquirmtcnts  bmcd  upon  job  demands, 
AFHRL'.TR^TSaS,  AD-AOlS  756.  Lacto^  AFB,  TX:  Oceupaltonal  and  Mani>ower  Research 
DiMsion,Ajr  Force  Ihiman  Resourcai  LahDrato^ 

Christal,  REm  A  ^^mv^^^^.  N(bw  CODAP  program  for  Qnafytini  tmk  fhctor  informtioti 
AF[lRl^TR^76^3,  AD-AOas  12L  Uckland  AFB|  TK;  Occupationnraiia  Manpower  Reseafch 
Divtsioni  Air  Force  Hunian  Resources  Labprator^^  Way  1^ 

Lindqiibti  E.G;  J}ff5/|7?  mul  mmlvsis  of  experiments  in  psychDhgy  aiirf  education  Bomni  Houghton 

;7;;;;;^^^Mif(Ti!u;i953,:359^36K       -^-^--^^^^--.^-.^^^^^^^^  .....^  -.^^^ 

Watson^  W.J,,  d  Ciood^j  K.  Matching  fob  education  ^cquirmtmts  with  can^ 

A  piloi mothodolosical stiidy.  AFHRL-m*7S-79,  AD-A025  214/Lacklatid  AFB.TX^  Occupational 
and  Manpower  Research  iMvislon,  Air  Force  Human  Resources  Laboratory^  December  1975, 
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APfENDJX  A    INSTRUGTIONS  USED  BY .  JUDGES  WHEN  RATING  TASKS  IN 
MEDICAL  SERVICES  SPECIALIST  INVlNTORy  ON  CONSEDUENCES 
OF  INADEQUArE  PERTOWt^Cl 


E^plan&tlcn  '  . 

This  booklet  contains  a  listing  of  tasks  performed  In  your  career 
ladder*    Ipu  are  asked  to  rate  each  task  to  indicate  the  Probable 
ConsequencBS  of  Inadequate  Perforisance  of  the  task*     In  the  Air  Forcej 
tha  conie^^uences  of  ^inadequatt  perforfliaiice  of  aoine  tasks  are  much  more 
serious  thuX^  for  other  tasks*    For  eKampler  if  itiadequate  perf oririance 
of  a  task  wilL  almost  certainly  cause  an  aircraft  to  crash ^  or  a 
warehouse       bum  down^  or  an  almian  to  die,  this  would  be  more  serious 
than  inadequate  performance  of  a  task  which  merely  causes  Inconveiiience 
and  Ixritatioa,    As  another  exampLe,  the  probable  consequences  of 
inadequate  performajice  In  respondliig  to  a  fire  alann  wuld  be  much  more 
serious  tfia^  the  probable  consequences  of  tnadeqiaate  performaace  in 
folding  hospital  linen,  . 

Definition 

'  CDnsaquences  of  Inadequate  Performance  is  a  measure  of  the  serlousneas 
of  the  prob^bLe  consequences  of  Inadequate  performance  of  a  task.    It  Is 
measiired  In  terms  of  possible  injury  or  deaths  wasted  supplies  s  d^aged 
equlptnent^  pasted  man-hours  of  work,  etCt 

Your  Task 

Tjalng  che  rating  scale  below,  assign  a  numerical  rating  to  each  task 
In  this  boolclet  which  you  feel  describes  the  probahle  'consequemcea  of 
Inadequate  perf ormance  of  the  task*    Make  your  ratings  by  sliiiply  writing 
a  numbar  i  through  9  in  the  column  to  the  right  of  each  task.  Please 
attempt  to  rate  all  tasks. 

Rating  SQmlm 

If  th^  taak  Is  not  done  correctlyi  the  probable  consequencea  of 
inadequ^ite  perionnance  would  hei 


I. 

Wtntmal  (inadequate  peTformance  has  ini'ntmal  consectueticea) 

2. 

SLight  ; 

3. 

Wot  very  aerloua 

^1. 

Taltrly  serious 

S, 

Serious                                                                                ''  ; 

6. 

Very  seilous 

7. 

EKtranitly  serioxis                            .                                  ^  ,,. 

B. 

.  i^lmost  dlsaatBous                                                               .  „  ,„ v„,; 

9. 

Dliastrous  Cinfldequate  peirfoniiaiice  has  dlaaatroiis  consequences) 

'Your  efforts  in  cDmpleting  this  booklet  will  be  slnceroly  appreciated*  l^en" 
you  have  ftnlshed  ysur  ratings,  please  return  this  booklet  to  ^our  CBPO/DPtCPG, 


/imi/VD/.V  /i.    RAM  RATINGS  FDR  THREE  1YPICAL  "GOOD"  RATERS 


Rater  Number  15 

411322213034542434332432  33223423222433333232224443314334335334 
433343333433533433323343433333223333332i-i3343334434333333335333343222 
426343443346664554333335454466665  3333332366345555555553656654^ 
777773377764553444444556565544566775665555555556655554553343334333443 
5475544365485556779  44456654444333954654539465554343357754444444443334 
444434322344334332263344232234669  777577525456569845945497496669064635 
554454433345333473364755475433443766634555553333335476976645594455665 
5965577545556644993568 


Rater  NTumber.  44 

212455534543551445004332455224141544455211333355641144535351253545515 
435554555353555555333233333222213553452433353344566333333333366363343 
337555654457774844644367566456783665666344575554465557585445355333555 
657583455565444333456466534544665654466665734567777466553355565665752 
44763333664656567675435657576544385454456846885445  337666565S666'552265 
66535544446655553345445545  4446767555466455555567858845457577368344345 
465554555556355653565555455554564766675555787777777586777556  786655588 
5955567855557757993357 


Hater  Number  53 

3231 3242254354343741 3472346345256  35545  44334332447316445565735736MiA.4- 
5556547556527446355445568445445647774628666666554765554655555675^55^^^ 
6586667666689968558555665885 89993675555468935776697695555555355655668 
7799957889855767668878888788889888877788878788887777887866677787 
669777758858667888777856665888555975677879578787765688767787777774587 
777788555677776752775258563346888988566566687889878887789788889776678 
78777566677637788378887768685778586665765585766 7878785975685395877887 
7977589545344777785577 
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Rater  Numbe'r  85 

111  11111111  11  111  111122121 131 11 111 11 11 1111  lin-l  12111211111111111111111 
111111111111111111221111111111111111111111111111111111111111111111111 
111  11111111  11111111111111  1111111111111111111111111,1111111111111111111 
111  1111111111111  111111111  11111111111111111111111  111111111111111111111 
111  11111111  11111111111  111  11111111111111111111111111111111111111111111 
11111111111 111111111111111111111111111111111111111111  1111111111111111 
111  11111111  11  111111111  111  11 1111111111111111111  11 11111111 1111111111111 
1111111111111111111111 


Rater  Number  41  ■ 

222111  11 11U1  121  122291  111  111  1399111  11  8111 11  111  11 11  111  1111122322222222 
232111111221111111111111112111111111111111111111111111111111111111111 
11111111111 11  111  111111 1122212222112231132111132122211 1112222122212122 
1232211111111111111111111111111111111122111111 11111111111111112112122 
1232123122221  11122212211222211  1211111  1111  11111  11  11 1 11  111  1 1 1 11 1 1 1 12212 
1111111222221331  111321111 iri  11  13233332221421112211 13333222111221 11111 
111  11111111  11111111111  111  111111111111111111111111111111111111  11111111 
1111111111111111111111 


Rater  Number  87 

999999999999999999999999999999999999999999999999599999999 
999999999999999999999999999999999999999999999999999999999999999999999 
9999999999999999999999999999999999999999999999999999999999999 
999999999999999999999999999999999999999999999999999999999999999999999 
999999999999999999999999999999999999999999999999999999999999999999999 
9999999999999999999999  99999999999999999999999999999999999999999999999 
99999999999  99999999999999  999g9999999999999»'jg999999g99999999999999999 
9999999999999999999999 
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Rater  Number  22 


415275221471971871112251  1721611312273261  1.1624211111156666584475748324 
5AA477557475844477447566242222225572642646421226212421 1124611  111  13142 
111717555555221222111111311211524557252271511171122112113112116111111 
1T211111J1121 15243331224342535221311 134442142442555231111 117541525112 
431151 11 i5152521 35141  111  2255555225741 11 35541 221  1572223222191244421152 
111  11911 11861  17171814115151111111  11131124551116111  11  11111216662266617 
1 1 465471  111  111111811111111111111  1311121  111111121112111131111181122111 
1111124411111111151111 


Rater  Nuinber  88 


6&6755566676556667877679 1555656798767555666789777998989888998777676 79 
$678^7777888977766665055670000907000000000000000000000000000000000000 
OOOOOOOOOOOOOOOOOOOOOOOOOOOOQOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO 
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO 
OOOOOOOOOOOOOOOOOOOOOOOQQQOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO 
OOOOOOOQOOOOOOOOOOOOOOOOOOOOOOOOOOObOOOOOOOOOOOOOOOOOOOOOQOOOOOOOOOOO 
OWOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO 
OOOOOOOOOOOOOOOOOOOOOO 


Rater  Humber  61 


721475436984991474497772474655466648797971665899999794896683692755529 
9«»555S55555SSS55555555555555555555555555555555S555555555555555.5 
5$5S5555555555555555555555555555555555S55555555S5555555555555S55S5555 
S55S555555555'555555S55555555555555555555555555S5555555555555555555555 
555355555555555555555555555555555555555555555555555555555555555155555 
S5555555555555555555555555555555555555555555555555555555555555555535S 
5$5SS55555555S55S55555555555555555555555555555555S555555S55555555 
55555555555555555 
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APPEI^DIX  D.  RATINGS.  FOR,  TWO  BORDERLINE  CASES 


Rater  Number  16 


7578845569 789 849 776688747786664678857554446778?7864776a68876785878978 
876887787868987778678865878999897988997977788889899999899999899999999 
99998999989899998798889989889899889888877789777779  8888887758577566577 
78989 456687778654456546563354447566547665586866887 76444546S4557554 
53746532672733645545465646 466755377556455957886645 448875457445445 
66755  74455  74555443454355353334646777556455674745755533464475557443334 
54444346  45443443434443"454U6454643445355454434455544S55645 

4736245634554334563455  . 


Rater  Number  54 


9477955699969958897779979897995888599998959778989887799  88997987999999 
955999889999889799976999789551415557995799999987999455599986699999498 
9912229966699999999996699999999997999999999999999999999  99999999  799999 
9999999999999999999992225592999999999977957777779997777  77799799997779 
999995559999999999979999999975999999999997999999995999895999999995999 
995777779997999975599896525299899999799999929999995955959917593259945 
999999997779195995995999999999994959999799999999999989772999990212291 
5677224922299927999298 
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